LION KING MOVIE : Reviews Sentiment Analysis
Ashutosh Parida
convention followed in this notebook¶
Green cell: Graph/Analysis topic to be done
Blue cell: Inference from above graph/analysis
Importing Required Packages¶
In [243]:
# Usual packages
import os
import json
import requests
import datetime
import time
import joblib
import pandas as pd
import numpy as np
import warnings
warnings.filterwarnings("ignore")
from pandas.io.json import json_normalize
from itertools import chain
In [239]:
# graph related packages
import cufflinks as cf
import plotly.offline
import matplotlib.pyplot as plt
cf.go_offline()
cf.set_config_file(offline=False, world_readable=True)
from yellowbrick.text import TSNEVisualizer
from sklearn.feature_extraction.text import TfidfVectorizer
from wordcloud import WordCloud
from PIL import Image
from os import path
In [30]:
# text related packages
import spacy
# Load the language model
nlp = spacy.load('en_core_web_lg')
from spacy.lang.en.stop_words import STOP_WORDS
import textblob
from textblob import TextBlob
import re
In [117]:
# need for topic modelling
from sklearn import metrics
from sklearn.cluster import KMeans
from scipy.spatial.distance import cdist
from sklearn.decomposition import TruncatedSVD
from sklearn.feature_extraction.text import TfidfVectorizer
In [5]:
headers = {
'Referer': 'https://www.rottentomatoes.com/m/the_lion_king_2019/reviews?type=user',\
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, likeGecko) Chrome/74.0.3729.108 Safari/537.36',\
'X-Requested-With': 'XMLHttpRequest',\
}
url = 'https://www.rottentomatoes.com/napi/movie/9057c2cf-7cab-317f-876f-e50b245ca76e/reviews/user'
payload = {
'direction': 'next',
'endCursor': '',
'startCursor': ''
}
In [ ]:
### web scrapping Method
parent_path =os.getcwd()
## Storing all the json data in movie_rdata dir
reviews_path = parent_path +"\\movie_rdata"
os.chdir(reviews_path)
s = requests.Session()
i=0
while (i < 302):
time.sleep(5)
data=''
r=''
print(payload,"i=",i)
r = s.get(url, headers=headers, params=payload) # GET Call
data = r.json()
#print(data)
if(data['pageInfo']['hasNextPage']):
next_endCursor=data['pageInfo']['endCursor']
payload = {
'direction': 'next',
'endCursor': next_endCursor,
'startCursor': ''
}
filename="page"+str(i)+".json"
with open(filename, 'w') as json_file:
json.dump(data, json_file)
i=i+1
json_file.close()
In [249]:
# Process data from the collected Dir and make a df
print(reviews_path)
total_files_processed=0
for review_file in os.listdir(reviews_path):
if review_file.endswith(".json"):
with open(review_file) as infile:
#print(review_file)
jdata=json.load(infile)
if(total_files_processed>0):
movie_df=movie_df.append(json_normalize(jdata['reviews']))
#print("coming here",total_files_processed)
else:
movie_df=json_normalize(jdata['reviews'])
#print("first time",total_files_processed)
total_files_processed=total_files_processed+1
print("Total files processed=",total_files_processed)
os.chdir(parent_path)
In [14]:
# resetting index as each page index it as 0-9 from json collection method
movie_df=movie_df.reset_index(drop=True)
In [15]:
movie_df.head(12)
Out[15]:
In [16]:
movie_df.head()
Out[16]:
Adding target variable to the data¶
In [18]:
#### Rating score >3 is considered positive (0) and negative (1) otherwise
In [19]:
movie_df['targetSentiment']=[0 if x>3 else 1 for x in movie_df['score']]
In [20]:
pd.DataFrame(movie_df,columns=['targetSentiment','score']).head()
Out[20]:
EDA¶
In [21]:
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print(movie_df.info())
print("---+++----+++------+++----+++---SHAPE---+++----+++------+++----+++------+++----+++---+++")
print(movie_df.shape)
print("---+++----+++------+++----+++---COLUMNS---+++----+++------+++----+++----+++------+++----+++")
print(movie_df.columns)
In [42]:
movie_df['createDate']=pd.to_datetime(movie_df['createDate'])
movie_df['updateDate']=pd.to_datetime(movie_df['updateDate'])
In [22]:
movie_df.info()
In [23]:
movie_df.isnull().sum()
Out[23]:
In [24]:
print("---+++----+++------+++----+++---Null Inference---+++----+++------+++----+++------+++----+++---+++")
print("Accounlink/displayName/ImageUrl/AccountLink is having NULL values, which can be expected.")
EDA of non-text related data¶
Target sentiment class division
In [33]:
HAPPY=0
SAD=1
movie_df.groupby('targetSentiment').score.count().iplot(
kind='bar',
barmode='group',
title='Sentiment - Positive vs Negative ',
linecolor='black',
xTitle='Sentiment',
yTitle='# Reviewers'
)
pos_users=movie_df.groupby('targetSentiment').score.count()[0]
neg_users=movie_df.groupby('targetSentiment').score.count()[1]
pos_neg_split=np.round((neg_users/(neg_users+pos_users)),4)*100
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print("% Of people who did not like this movie",pos_neg_split)
Inference:
Class imbalance seen for target variable!
So precautions needs to be taken to handle the train data for modelling
Class imbalance seen for target variable!
So precautions needs to be taken to handle the train data for modelling
Rating scores distribution
In [38]:
movie_df.score.value_counts().iplot(
kind='bar',
xTitle='rating',
linecolor='black',
yTitle='# Reviewers',
title="Reviewer's Rating Distribution")
Inference:
Majority of the reviewers have rated it : 5
And, ratings suggest only 28% people have dis-liked the movie
Majority of the reviewers have rated it : 5
And, ratings suggest only 28% people have dis-liked the movie
Ratings given by Verified Users and Super Users
Verified Users: Users who bought tickets
Super Users : Users who have been regular and have thus attained special class
Verified Users: Users who bought tickets
Super Users : Users who have been regular and have thus attained special class
In [39]:
verifiedr_ratings = movie_df[movie_df['isVerified']==1]['targetSentiment'].value_counts()
not_verifiedr_ratings = movie_df[movie_df['isVerified']==0]['targetSentiment'].value_counts()
superr_ratings = movie_df[movie_df['isSuperReviewer']==1]['targetSentiment'].value_counts()
not_superr_ratings = movie_df[movie_df['isSuperReviewer']==0]['targetSentiment'].value_counts()
df1 = pd.DataFrame([verifiedr_ratings,not_verifiedr_ratings])
df2 = pd.DataFrame([superr_ratings,not_superr_ratings])
df1.index = ['Verified User','not Verified User']
df2.index = ['Super User','not Super User']
df1.iplot(kind='bar',barmode='stack',title='Ratings by Verified User Type')
df2.iplot(kind='bar',barmode='stack',title='Ratings by Super User Type')
Inference:
% Verified Users who dis-liked the movie: 24.7
And, only 1 Super reviewer saw the movie
% Not-Super users who dis-liked the movie: 27.5
% Verified Users who dis-liked the movie: 24.7
And, only 1 Super reviewer saw the movie
% Not-Super users who dis-liked the movie: 27.5
Ratings(type) Trend across the timeline(days)
In [44]:
movie_trend=movie_df.copy(deep=True)
movie_trend.index = movie_trend['updateDate']
In [45]:
movie_trend.resample('D').mean().score.iplot(
kind='bar',
bins=50,
xTitle='Date',
linecolor='black',
yTitle='mean(Rating)',
title='Daywise Rating Distribution')
Inference:
Average rating across the days is : 4
So the movie on an average can be considered going good with the audience
Average rating across the days is : 4
So the movie on an average can be considered going good with the audience
Ratings(#) Trend across the timeline(days)
In [46]:
movie_trend.resample('D').count().score.iplot(
kind='bar',
bins=50,
xTitle='Date',
linecolor='black',
yTitle='# Ratings',
title='Rating Traffic Trend')
Inference:
People see movies over weekend and more traffic on Sunday being an universal off. Aug 3 and 4 are weekend. Hence the no. of reviews are high on Monday i.e. Aug 5
Similar trend was seen on next weekend, with ratings steadily rising near weekend and reaching local maximum on Monday, as the next day feed from users
People see movies over weekend and more traffic on Sunday being an universal off. Aug 3 and 4 are weekend. Hence the no. of reviews are high on Monday i.e. Aug 5
Similar trend was seen on next weekend, with ratings steadily rising near weekend and reaching local maximum on Monday, as the next day feed from users
Analysing Profanity and Spoiler and its relation with rating¶
In [54]:
print(movie_df.hasSpoilers.value_counts())
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print("\n% Of Users using Spoilers",round(movie_df.hasSpoilers.value_counts()[1]/(movie_df.hasSpoilers.value_counts()[0] +\
movie_df.hasSpoilers.value_counts()[1]),4))
In [53]:
print(movie_df.hasProfanity.value_counts())
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print("\n% Of Users using Profanity",round(movie_df.hasProfanity.value_counts()[1]/(movie_df.hasProfanity.value_counts()[0] +\
movie_df.hasProfanity.value_counts()[1]),4))
Profanity relation with Ratings
Profanity trend across Timeline(days)
In [49]:
movie_trend[movie_trend['hasProfanity']==True].score.sort_index(axis=0).iplot(
kind='bar',
xTitle='Date',
linecolor='black',
yTitle='Ratings',
title='Profanity vs Rating')
Inference:
Profanity has been used throught out the date range.There is no correlation with specific dates
All the reviews have been covered with Profanity. Its not inclined for specific rating.No 1:1 mapping
Analysis of profanity text will help in understanding the terms to keep for sentiment analysis. But here, percentage of users using profanity is very less.
Profanity has been used throught out the date range.There is no correlation with specific dates
All the reviews have been covered with Profanity. Its not inclined for specific rating.No 1:1 mapping
Analysis of profanity text will help in understanding the terms to keep for sentiment analysis. But here, percentage of users using profanity is very less.
Profanity traffic across Ratings
In [50]:
movie_trend[movie_trend['hasProfanity']==True].score.sort_index(axis=0).value_counts().iplot(
kind='bar',
xTitle='rating',
linecolor='black',
yTitle='count',
title='#Rating Distribution For Profanity reviews')
Inference:
People can use bad/swear words showing displeasure as well as excitement
People can use bad/swear words showing displeasure as well as excitement
In [52]:
perc_prof_reviews=round(movie_df.hasProfanity.value_counts()[1]/(movie_df.hasProfanity.value_counts()[0] +\
movie_df.hasProfanity.value_counts()[1]),4)
perc_prof_reviews=round((movie_df.hasProfanity.value_counts()[1]/movie_df['user.userId'].count())*100,2)
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print("% of Users using profanity :"+ str(perc_prof_reviews)+"\nThis is very less. Can be ignored.")
EDA for text¶
Making copy of data df and dropping unnecessary columns¶
In [55]:
total_data=movie_df.copy(deep=True)
In [56]:
total_data.columns.values
Out[56]:
In [57]:
columns_except_review_score=['createDate', 'displayImageUrl', 'displayName', 'hasProfanity',
'hasSpoilers', 'isSuperReviewer', 'isVerified', 'rating','timeFromCreation', 'updateDate', 'user.accountLink',
'user.displayName', 'user.realm', 'user.userId']
In [58]:
total_data.drop(columns_except_review_score,inplace=True,axis=1)
In [59]:
total_data.head()
Out[59]:
Plotting graphs BEFORE any cleaning of text¶
Word Count trend of Reviews
In [60]:
total_data['word_count'] = total_data['review'].apply(lambda x: len(str(x).split()))
In [237]:
total_data['word_count'].iplot(
kind='violin',
xTitle='review length',
linecolor='black',
yTitle='count',
title='Word count distribution before cleaning text',colors='#604d9e')
Inference:
Median word count usage shows : 15
Median word count usage shows : 15
Review length of Reviews
In [236]:
total_data['review_len'] = total_data['review'].astype(str).apply(len)
total_data['review_len'].iplot(
kind='violin',
xTitle='review length',
linecolor='black',
yTitle='count',
title='Review Length Distribution before cleaning text')
Inference:
Median Review length usage shows : 84
Median Review length usage shows : 84
In [63]:
print("---+++----+++------+++----+++---Avg(ReviewLen)/Avg(#Words)---+++----+++------+++----+++------+++----+++---+++")
print("\nReview Length is greater than #words in it by factor of:",\
round((total_data['review_len'].sum()/total_data['word_count'].sum()),2))
Analysis of extra length reviews/having max words
In [64]:
print("---+++----+++------+++----+++---#Word>700---+++----+++------+++----+++------+++----+++---+++")
print(total_data[total_data['word_count']>700])
print("---+++----+++------+++----+++---ReviewLength>4000---+++----+++------+++----+++------+++----+++---+++")
print(total_data[total_data['review_len']>4000])
In [65]:
print(movie_df.iloc[1854])
print("---+++----+++------+++----+++---INFERENCE---+++----+++------+++----+++------+++----+++---+++")
print("\nHe liked the movie, so wrote lengthy summary")
In [66]:
print(movie_df.iloc[1953])
print("---+++----+++------+++----+++---INFERENCE---+++----+++------+++----+++------+++----+++---+++")
print("\nHe dis-liked the movie, so wrote lengthy summary with some spoiler movie events!")
N-Grams analysis
Top Unigram words before stem/lemma¶
In [67]:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.feature_extraction.text import CountVectorizer
def get_top_n_unigram(corpus, n=None):
vec = CountVectorizer().fit(corpus)
bag_of_words = vec.transform(corpus)
sum_words = bag_of_words.sum(axis=0)
words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
return words_freq[:n]
common_words = get_top_n_unigram(total_data['review'], 45)
df_uni = pd.DataFrame(common_words, columns = ['unigrams' , 'count'])
df_uni.groupby('unigrams').sum()['count'].\
sort_values(ascending=False).iplot(kind='bar',yTitle='Count', linecolor='black',\
title='Top 45 Unigrams in review before cleaning')
In [68]:
def get_top_n_bigram(corpus, n=None):
vec = CountVectorizer(ngram_range=(2, 2)).fit(corpus)
bag_of_words = vec.transform(corpus)
sum_words = bag_of_words.sum(axis=0)
words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
return words_freq[:n]
common_words = get_top_n_bigram(total_data['review'], 45)
df_bi = pd.DataFrame(common_words, columns = ['bigrams' , 'count'])
df_bi.groupby('bigrams').sum()['count'].\
sort_values(ascending=False).iplot(kind='bar',yTitle='Count', linecolor='black',\
title='Top 45 Bigrams in review before cleaning')
In [69]:
def get_top_n_trigram(corpus, n=None):
vec = CountVectorizer(ngram_range=(3, 3)).fit(corpus)
bag_of_words = vec.transform(corpus)
sum_words = bag_of_words.sum(axis=0)
words_freq = [(word, sum_words[0, idx]) for word, idx in vec.vocabulary_.items()]
words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)
return words_freq[:n]
common_words = get_top_n_trigram(total_data['review'], 45)
df_tri = pd.DataFrame(common_words, columns = ['trigrams' , 'count'])
df_tri.groupby('trigrams').sum()['count'].\
sort_values(ascending=False).iplot(kind='bar',yTitle='Count', linecolor='black',\
title='Top 45 Trigrams in review before cleaning')
Inference:
N-grams is getting biased with presence of stop-words.
e.g. Top n-grams in 3 graphs shows : the (uni) --> the original (bi) --> to the original (tri)
Since the majority of reviews is positive sentiment, top-45 n-grams in all category shows positive words
N-grams is getting biased with presence of stop-words.
e.g. Top n-grams in 3 graphs shows : the (uni) --> the original (bi) --> to the original (tri)
Since the majority of reviews is positive sentiment, top-45 n-grams in all category shows positive words
### Plotting graph **AFTER** cleaning of text
Cleaning the review text¶
In [71]:
#print(list(STOP_WORDS))
from itertools import compress
from contractions import CONTRACTION_MAP
import unicodedata
def expand_contractions(text, contraction_mapping=CONTRACTION_MAP):
contractions_pattern = re.compile('({})'.format('|'.join(contraction_mapping.keys())),
flags=re.IGNORECASE|re.DOTALL)
def expand_match(contraction):
#print(contraction)
#print(type(contraction))
match = contraction.group(0)
#print(match)
first_char = match[0]
#print(first_char)
expanded_contraction = contraction_mapping.get(match)\
if contraction_mapping.get(match)\
else contraction_mapping.get(match.lower())
expanded_contraction = first_char+expanded_contraction[1:]
return expanded_contraction
#print(f"The expand match is {expand_match} and text is {text}")
expanded_text = contractions_pattern.sub(expand_match, text)
expanded_text = re.sub("'", "", expanded_text)
return expanded_text
def normalize_accented_chars(text):
text = unicodedata.normalize('NFKD', text).encode('ascii', 'ignore').decode('utf-8', 'ignore')
#https://docs.python.org/2/library/unicodedata.html
return text
def add_NOT_after_not(sentence=None):
transformed = re.sub(r'\b(?:not|never|no)\b[\w\s]+[^\w\s]',
lambda match: re.sub(r'(\s+)(\w+)', r'\1NOT_\2', match.group(0)),
sentence,
flags=re.IGNORECASE)
return transformed
def tokenize_lemma_clean(dataframe,column_name):
# text cleaning and pre-processing
for index,row in dataframe.iterrows():
#print(row[column_name],":::",row['score'])
row[column_name]=row[column_name].lower()
doc=nlp(row[column_name])
# remove stop words
clean_tokens1 = [token for token in doc if not token.is_stop]
# may use STOP words for conjuction detection
#remove words with <=2 chars
clean_tokens1 = [token for token in clean_tokens1 if len(token.text)>2]
#remove non-alpha
clean_tokens2_bool = [token.is_alpha for token in clean_tokens1]
#clean_tokens2_bool = [token.is_alpha for token in doc]
clean_tokens2=list(compress(clean_tokens1,clean_tokens2_bool))
#not_string = re.compile("not_*{7,}")
#use only lemma
clean_tokens3 = [token.lemma_ for token in clean_tokens2]
#print(clean_tokens3)
clean_text=' '.join(clean_tokens3)
#dataframe.at[index,column_name]=clean_text
dataframe.at[index,"clean_text1"]=clean_text
def pos_to_keep(dataframe,column_name,dest_column_name,x1=None,x2=None,x3=None,x4=None):
# text cleaning and pre-processing
for index,row in dataframe.iterrows():
#print(row['STORY'],":::",row['SECTION'])
doc=nlp(row[column_name])
# remove stop words
nn_tokens1 = [token.text for token in doc if ((token.text in ['like','love'])or(token.pos_ in [x1,x2,x3,x4]))]
nn_text=' '.join(nn_tokens1)
dataframe.at[index,dest_column_name]=nn_text
In [72]:
## keeping cleaned data in 'clean_text' column for safekeeping and comparing with original
In [73]:
total_data.insert(2,"clean_text","")
Spelling correction¶
In [ ]:
#total_data['clean_text'] = [TextBlob(text).correct() for text in total_data['review']]
Expand contractions¶
In [74]:
total_data['clean_text'] = [expand_contractions(text) for text in total_data['review']]
#total_data['clean_text'] = [expand_contractions(text) for text in total_data['clean_text']]
Normalize accented characters¶
In [75]:
total_data['clean_text'] = [normalize_accented_chars(text) for text in total_data['clean_text']]
Add NOT_ after not is encountered till punctuation¶
In [76]:
total_data['clean_text'] = [add_NOT_after_not(text) for text in total_data['clean_text']]
Lemmatization¶
In [79]:
## keeping the second stage of clean data in other column "clean_text1" for safekeeping and comparing with earlier clean data
## 'clean_text1' will have the final cleaned data after all these above process and lemmatization
In [78]:
tokenize_lemma_clean(total_data,"clean_text")
In [80]:
total_data.head(20)
Out[80]:
Plotting graphs¶
Word Count trend of Reviews
In [238]:
total_data['word_count_clean'] = total_data['clean_text1'].apply(lambda x: len(str(x).split()))
total_data['word_count_clean'].iplot(
kind='violin',
xTitle='review length',
linecolor='black',
yTitle='count',
title='Word count Distribution after cleaning text',colors='#604d9e')
Inference:
Median word count=6
Upper wishker = 24
so even if we take 3*whisker, so taking ~40 words we will cover 99.7% dataset
Median word count=6
Upper wishker = 24
so even if we take 3*whisker, so taking ~40 words we will cover 99.7% dataset
Review length of Reviews
In [234]:
total_data['review_len'] = total_data['clean_text1'].astype(str).apply(len)
total_data['review_len'].iplot(
kind='violin',
xTitle='review length',
linecolor='black',
yTitle='count',
title='Review Length Distribution after cleaning text')
Inference:
Median Review length usage shows : 42
i.e. drops to almost half from original un-cleaned review length
Median Review length usage shows : 42
i.e. drops to almost half from original un-cleaned review length
Top Unigram words after text cleaning¶
In [82]:
common_words = get_top_n_unigram(total_data['clean_text1'], 45)
#for word, freq in common_words:
# print(word, freq)
df_uni_clean = pd.DataFrame(common_words, columns = ['unigrams' , 'count'])
df_uni_clean.groupby('unigrams').sum()['count'].\
sort_values(ascending=False).iplot(kind='bar',yTitle='Count', linecolor='black',\
title='Top 45 Unigrams in review after text cleaning')
Top Bigram words text cleaning¶
In [83]:
common_words = get_top_n_bigram(total_data['clean_text1'], 45)
#for word, freq in common_words:
# print(word, freq)
df_bi_clean = pd.DataFrame(common_words, columns = ['bigrams' , 'count'])
df_bi_clean.groupby('bigrams').sum()['count'].\
sort_values(ascending=False).iplot(kind='bar',yTitle='Count', linecolor='black',\
title='Top 45 Bigrams in review after text cleaning')
Top Trigram words after text cleaning¶
In [84]:
common_words = get_top_n_trigram(total_data['clean_text1'], 45)
#for word, freq in common_words:
# print(word, freq)
df_tri_clean = pd.DataFrame(common_words, columns = ['trigrams' , 'count'])
df_tri_clean.groupby('trigrams').sum()['count'].\
sort_values(ascending=False).iplot(kind='bar',yTitle='Count', linecolor='black',\
title='Top 45 Trigrams in review after text cleaning')
Inference:
More meaningful words are displated now
Even tigrams now give a good sense of topic being discussed in reviews
Since most of the reviews are with positive sentiment, hence more positive topics seems to be discussed
More meaningful words are displated now
Even tigrams now give a good sense of topic being discussed in reviews
Since most of the reviews are with positive sentiment, hence more positive topics seems to be discussed
Word Cloud representation of cleaned review text
In [85]:
all_words = ' '.join([text for text in total_data['clean_text1']])
lion_mask = np.array(Image.open("lion.jpg"))
wordcloud = WordCloud(background_color="white",width=5000, height=5000, random_state=21,\
max_font_size=110,mask=lion_mask,contour_width=3, contour_color='steelblue').generate(all_words)
plt.figure(figsize=(100, 70))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis('off')
plt.show()
Word Cloud representation of cleaned review text but for only Negative Reviews
In [86]:
negative_sent_text=total_data[total_data['targetSentiment']==1]['clean_text1']
all_words = ' '.join([text for text in negative_sent_text])
lion_mask = np.array(Image.open("lion.jpg"))
wordcloud = WordCloud(background_color="white",width=5000, height=5000, random_state=21,\
max_font_size=110,mask=lion_mask,contour_width=3, contour_color='orange').generate(all_words)
plt.figure(figsize=(100, 70))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis('off')
plt.show()
Inference:
Nouns are replicated in both the pictures
Adverb,Adjective are the differenciating factors, hence will be a good feature for classification
Nouns are replicated in both the pictures
Adverb,Adjective are the differenciating factors, hence will be a good feature for classification
POS Tags Analysis¶
Distribution of different POS
In [88]:
#!pip install textblob
from textblob import TextBlob
blob = TextBlob(str(total_data['clean_text1'].tolist()))
pos_df = pd.DataFrame(blob.tags, columns = ['word' , 'pos'])
#pos_df = pos_df[~pos_df['word'].str.isnumeric()]
pos_df = pos_df.pos.value_counts()[:20]
pos_df.iplot(
kind='bar',
xTitle='POS',
yTitle='count',
title='Top 20 Part-of-speech tagging for review corpus')
Plotting the documents in vector space and classifying on target sentiment
In [241]:
tfidf = TfidfVectorizer()
docs = tfidf.fit_transform(total_data['clean_text1'])
tsne = TSNEVisualizer()
tsne.fit_transform(docs, total_data['targetSentiment'])
tsne.poof()
Keeping only adjective/adverb/verb in reviews and plotting the documents
In [93]:
pos_to_keep(total_data,"clean_text1","clean_adj_adv_verb_col","ADJ","ADV","VERB")
In [95]:
total_data.head()
Out[95]:
In [96]:
tfidf = TfidfVectorizer()
docs = tfidf.fit_transform(total_data['clean_adj_adv_verb_col'])
tsne = TSNEVisualizer()
tsne_results=tsne.fit_transform(docs, total_data['targetSentiment'])
tsne.poof()
Inference:
Documents are much more clearly separated with using Adj/Adv/verb
Separated islands and denser categories seen in comparison to earlier plot having all POS tags
This can be used in feature engineering later.
Documents are much more clearly separated with using Adj/Adv/verb
Separated islands and denser categories seen in comparison to earlier plot having all POS tags
This can be used in feature engineering later.
In [98]:
pos_to_keep(total_data,"clean_text1","clean_adj_col","ADJ")
In [242]:
total_data.head()
Out[242]:
In [100]:
tfidf = TfidfVectorizer()
docs = tfidf.fit_transform(total_data['clean_adj_col'])
tsne = TSNEVisualizer()
tsne.fit_transform(docs, total_data['targetSentiment'])
tsne.poof()
Inference:
Documents are much more clearly separated with using only Adj
More separation seen and more denser categories seen in comparison to earlier plot
POS tags/text like ADJ can be used in feature engineering later.
Documents are much more clearly separated with using only Adj
More separation seen and more denser categories seen in comparison to earlier plot
POS tags/text like ADJ can be used in feature engineering later.
Wordcloud of Adjectives in negative reviews
In [256]:
negative_sent_text_adj=total_data[total_data['targetSentiment']==1]['clean_adj_col']
all_words = ' '.join([text for text in negative_sent_text_adj])
lion_mask = np.array(Image.open("lion.jpg"))
wordcloud = WordCloud(background_color="white",width=5000, height=5000, random_state=21,\
max_font_size=110,mask=lion_mask,contour_width=3, contour_color='orange').generate(all_words)
plt.figure(figsize=(100, 70))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis('off')
plt.show()
# love,like - keeping as default as its common and needs to be there
Wordcloud of Adjectives in positive reviews
In [200]:
pos_sent_text_adj=total_data[total_data['targetSentiment']==0]['clean_adj_col']
all_words = ' '.join([text for text in pos_sent_text_adj])
lion_mask = np.array(Image.open("lion.jpg"))
wordcloud = WordCloud(background_color="white",width=5000, height=5000, random_state=21,\
max_font_size=110,mask=lion_mask,contour_width=3, contour_color='steelblue').generate(all_words)
plt.figure(figsize=(100, 70))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis('off')
plt.show()
# love,like - keeping as default as its common and needs to be there
Topic modelling¶
In [114]:
#define vectorizer parameters
tfidf_vectorizer = TfidfVectorizer(max_df=0.9, max_features=20000,
min_df=0.05,
use_idf=True, ngram_range=(1,4))
tfidf_matrix = tfidf_vectorizer.fit_transform(total_data['clean_text1'])
tfidf_vectorizer.get_feature_names()
n_components = 26
svd_model = TruncatedSVD(n_components=n_components, algorithm='randomized',n_iter=20,random_state=143)
svd_matrix = svd_model.fit(tfidf_matrix)
doc_topic_matrix = svd_matrix.transform(tfidf_matrix)
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print(f"tfidf_matrix.shape : {tfidf_matrix.shape}")
print(f"svd_matrix.n_components : {svd_matrix.n_components}")
print(f"\n\nExplained Variance Ratio : {svd_matrix.explained_variance_ratio_}")
print(f"\nTotal Explained Variance : {round(svd_matrix.explained_variance_ratio_.sum() * 100, 2)} %")
print(f"\nThe singular values are {svd_matrix.singular_values_}")
Picking the most important words in each topic¶
In [115]:
terms = tfidf_vectorizer.get_feature_names()
for i, comp in enumerate(svd_model.components_):
#print(f"The component is {comp} and shape is {comp.shape}")
terms_comp = zip(terms, comp)
sorted_terms = sorted(terms_comp, key= lambda x:x[1], reverse=True)[:6]
print("Topic "+str(i)+": ")
for t in sorted_terms:
print(t[0],end=" ")
print("\n")
Calculate optimal cluster of topics¶
In [118]:
Sum_of_squared_distances = []
K = range(1,8)
for k in K:
kmeanModel = KMeans(n_clusters=k, random_state=143)
kmeanModel.fit(doc_topic_matrix)
Sum_of_squared_distances.append(kmeanModel.inertia_)
In [119]:
font = {'family' : 'normal',
'weight' : 'bold',
'size' : 10}
plt.rc('font', **font)
plt.plot(K, Sum_of_squared_distances, 'bx-')
plt.xlabel('k')
plt.ylabel('Sum_of_squared_distances')
plt.title('Elbow Method For Optimal k')
plt.show()
Inference:
Since target class is imbalanced and mostly people are talking positive. Hence the higher freq of words are positive in most of the topics
So the elbow curve also shows a smooth curve for elbow point
Since target class is imbalanced and mostly people are talking positive. Hence the higher freq of words are positive in most of the topics
So the elbow curve also shows a smooth curve for elbow point
In [120]:
# Choosing the best possible elbow point
num_clusters = 4
km = KMeans(n_clusters=num_clusters)
km.fit(doc_topic_matrix)
clusters = km.labels_.tolist()
centers = km.cluster_centers_
print(f"the cluster centers are {centers}")
joblib.dump(km, 'doc_topic_cluster_best_K.pkl')
Out[120]:
In [121]:
clusters = km.labels_.tolist()
total_data['doc_topic_cluster_group'] = clusters
In [244]:
total_data.head()
Out[244]:
In [128]:
doc_cluster_df = pd.DataFrame(total_data)
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print("No. of docs in each cluster id")
doc_cluster_df['doc_topic_cluster_group'].value_counts()
Out[128]:
Fetching the most frequent words among each cluster¶
In [131]:
from itertools import chain
doc_cluster_df['tokenized_text'] = [text.split(' ') for text in doc_cluster_df['clean_text1']]
grouped_text = doc_cluster_df.groupby('doc_topic_cluster_group')['tokenized_text']
frequent_words_df = pd.DataFrame(columns={"values", "counts", "cluster_id"})
for num in range(num_clusters):
values, counts = np.unique(list(chain.from_iterable(grouped_text.get_group(num))), return_counts=True)
sorted_indices = np.argsort(-counts)
frequent_words_df = frequent_words_df.append({"values":values[sorted_indices], "counts":counts[sorted_indices], "cluster_id": num}, ignore_index=True)
In [132]:
frequent_words_df.head()
Out[132]:
In [150]:
font = {'family' : 'normal',
'weight' : 'bold',
'size' : 35}
plt.rc('font', **font)
fig = plt.figure(figsize=(20,50))
plt.subplot(2,2,1)
plt.xlabel("Topic 0",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[0,'values'][:20], frequent_words_df.loc[0,'counts'][:20])
plt.gca().invert_yaxis()
plt.subplot(2,2,2)
plt.xlabel("Topic 1",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[1,'values'][:20], frequent_words_df.loc[1,'counts'][:20])
plt.gca().invert_yaxis()
plt.subplot(2,2,3)
plt.xlabel("Topic 2",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[2,'values'][:20], frequent_words_df.loc[2,'counts'][:20])
plt.gca().invert_yaxis()
plt.subplot(2,2,4)
plt.xlabel("Topic 3",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[3,'values'][:20], frequent_words_df.loc[3,'counts'][:20])
plt.gca().invert_yaxis()
Inference:
Since target class is imbalanced and mostly people are talking positive. Hence, the higher freq of words are positive in most of the topics
Thus topic are revolving around all these positive words
e.g.
Topic 0 : Loved the remake of Original Lion King, animal animation and voice over is amazing
Topic 1 : Family movie, visual and graphics loved by kids
Since target class is imbalanced and mostly people are talking positive. Hence, the higher freq of words are positive in most of the topics
Thus topic are revolving around all these positive words
e.g.
Topic 0 : Loved the remake of Original Lion King, animal animation and voice over is amazing
Topic 1 : Family movie, visual and graphics loved by kids
Topic modelling for Negative Reviews¶
In [179]:
#define vectorizer parameters
tfidf_vectorizer = TfidfVectorizer(max_df=0.9, max_features=20000,
min_df=0.05,
use_idf=True, ngram_range=(1,4))
neg_senti_df=total_data[total_data['targetSentiment']==1].copy(deep=True)
tfidf_matrix = tfidf_vectorizer.fit_transform(neg_senti_df['clean_text1'])
tfidf_vectorizer.get_feature_names()
n_components = 26
svd_model = TruncatedSVD(n_components=n_components, algorithm='randomized',n_iter=20,random_state=143)
svd_matrix = svd_model.fit(tfidf_matrix)
doc_topic_matrix = svd_matrix.transform(tfidf_matrix)
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print(f"tfidf_matrix.shape : {tfidf_matrix.shape}")
print(f"svd_matrix.n_components : {svd_matrix.n_components}")
print(f"\n\nExplained Variance Ratio : {svd_matrix.explained_variance_ratio_}")
print(f"\nTotal Explained Variance : {round(svd_matrix.explained_variance_ratio_.sum() * 100, 2)} %")
print(f"\nThe singular values are {svd_matrix.singular_values_}")
In [183]:
terms = tfidf_vectorizer.get_feature_names()
for i, comp in enumerate(svd_model.components_):
#print(f"The component is {comp} and shape is {comp.shape}")
terms_comp = zip(terms, comp)
sorted_terms = sorted(terms_comp, key= lambda x:x[1], reverse=True)[:6]
print("Topic "+str(i)+": ")
for t in sorted_terms:
print(t[0],end=" ")
print("\n")
In [184]:
Sum_of_squared_distances = []
K = range(1,8)
for k in K:
kmeanModel = KMeans(n_clusters=k, random_state=143)
kmeanModel.fit(doc_topic_matrix)
Sum_of_squared_distances.append(kmeanModel.inertia_)
font = {'family' : 'normal',
'weight' : 'bold',
'size' : 10}
plt.rc('font', **font)
plt.plot(K, Sum_of_squared_distances, 'bx-')
plt.xlabel('k')
plt.ylabel('Sum_of_squared_distances')
plt.title('Elbow Method For Optimal k')
plt.show()
In [186]:
# Choosing the best possible elbow point
num_clusters = 5
km = KMeans(n_clusters=num_clusters)
km.fit(doc_topic_matrix)
clusters = km.labels_.tolist()
centers = km.cluster_centers_
print(f"the cluster centers are {centers}")
joblib.dump(km, 'doc_topic_cluster_best_K1.pkl')
clusters = km.labels_.tolist()
neg_senti_df['doc_topic_cluster_group'] = clusters
In [187]:
doc_cluster_df = pd.DataFrame(neg_senti_df)
print("---+++----+++------+++----+++---INFO---+++----+++------+++----+++------+++----+++---+++")
print("No. of docs in each cluster id")
doc_cluster_df['doc_topic_cluster_group'].value_counts()
Out[187]:
In [188]:
doc_cluster_df['tokenized_text'] = [text.split(' ') for text in doc_cluster_df['clean_text1']]
grouped_text = doc_cluster_df.groupby('doc_topic_cluster_group')['tokenized_text']
frequent_words_df = pd.DataFrame(columns={"values", "counts", "cluster_id"})
for num in range(num_clusters):
values, counts = np.unique(list(chain.from_iterable(grouped_text.get_group(num))), return_counts=True)
sorted_indices = np.argsort(-counts)
frequent_words_df = frequent_words_df.append({"values":values[sorted_indices], "counts":counts[sorted_indices], "cluster_id": num}, ignore_index=True)
In [189]:
frequent_words_df.head()
Out[189]:
In [192]:
font = {'family' : 'normal',
'weight' : 'bold',
'size' : 35}
plt.rc('font', **font)
fig = plt.figure(figsize=(20,50))
plt.subplot(3,2,1)
plt.xlabel("Topic 0",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[0,'values'][:20], frequent_words_df.loc[0,'counts'][:20])
plt.gca().invert_yaxis()
plt.subplot(3,2,2)
plt.xlabel("Topic 1",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[1,'values'][:20], frequent_words_df.loc[1,'counts'][:20])
plt.gca().invert_yaxis()
plt.subplot(3,2,3)
plt.xlabel("Topic 2",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[2,'values'][:20], frequent_words_df.loc[2,'counts'][:20])
plt.gca().invert_yaxis()
plt.subplot(3,2,4)
plt.xlabel("Topic 3",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[3,'values'][:20], frequent_words_df.loc[3,'counts'][:20])
plt.gca().invert_yaxis()
plt.subplot(3,2,5)
plt.xlabel("Topic 4",fontsize=24,color='steelblue',fontfamily='serif',weight='bold')
plt.barh(frequent_words_df.loc[4,'values'][:20], frequent_words_df.loc[4,'counts'][:20])
plt.gca().invert_yaxis()
Inference:
considering negative reviews, except Topic-0, which is mostly skewed towards positive, others are have negative elements.
Possible reasons for Topic-0 having positive words is because of people at times consider 3 as average/positive ratings, and we have marked it negative. Another possible reasons may be thwarted sentences.
As is seen in other topics, it seems people are talking about it being a remake of orginal movie and it lacks in some scenes, story, at times lack emotions.
e.g.
Topic 0 : Remake has Disney animation characters with live animal feel
Topic 1 : Liked original, but remake lacks emotion.
considering negative reviews, except Topic-0, which is mostly skewed towards positive, others are have negative elements.
Possible reasons for Topic-0 having positive words is because of people at times consider 3 as average/positive ratings, and we have marked it negative. Another possible reasons may be thwarted sentences.
As is seen in other topics, it seems people are talking about it being a remake of orginal movie and it lacks in some scenes, story, at times lack emotions.
e.g.
Topic 0 : Remake has Disney animation characters with live animal feel
Topic 1 : Liked original, but remake lacks emotion.
Sentiment polarity analysis at review level¶
In [245]:
def review_polarity(dataframe,column_name,dest_column_name):
for index,row in dataframe.iterrows():
review=TextBlob(row[column_name])
dataframe.at[index,dest_column_name]=review.polarity
In [206]:
review_polarity(total_data,"clean_text1","rev_polarity")
In [228]:
total_data.head()
Out[228]:
In [210]:
sent_polarity_df=pd.DataFrame(total_data,columns=['targetSentiment','rev_polarity'])
In [225]:
sent_polarity_df[sent_polarity_df['targetSentiment']==0]['rev_polarity'].iplot(
kind='hist',
xTitle='rating',
linecolor='black',
yTitle='count',
color="blue",
title='Polarity distribution of Positive reviews')
In [226]:
sent_polarity_df[sent_polarity_df['targetSentiment']==1]['rev_polarity'].iplot(
kind='hist',
xTitle='rating',
linecolor='black',
yTitle='count',
color="orange",
title='Polarity distribution of Negative reviews')
Inference:
Polarity distribution of Positive reviews have less mis-classification, than negative reviews. For negative reviews, there is high no. of misclassification happening in 0-0.5 polarity range
Polarity distribution of Positive reviews have less mis-classification, than negative reviews. For negative reviews, there is high no. of misclassification happening in 0-0.5 polarity range
keeping a copy of total modifications and making a new df for going ahead of feature engineering¶
In [254]:
final_data=total_data.copy(deep=True)
In [255]:
final_data.columns.values
Out[255]:
Ending EDA
In [ ]: